Skip to content

fix(review-memory): enforce structured output#4090

Open
alex-alecu wants to merge 5 commits into
mainfrom
review-memory-json-output
Open

fix(review-memory): enforce structured output#4090
alex-alecu wants to merge 5 commits into
mainfrom
review-memory-json-output

Conversation

@alex-alecu

Copy link
Copy Markdown
Contributor

Summary

Review Memory now asks the model for schema-checked JSON instead of trying to recover JSON from plain text. Bad or incomplete replies fail before any GitHub branch or pull request is created, while safe request details make failures easier to trace without storing prompts or model output. Structured-output routing is applied only where the selected provider supports it.

Verification

  1. Approve a Review Memory proposal with a valid model reply and confirm the REVIEW.md pull request opens.
  2. Return malformed or cut-off model output and confirm the proposal becomes retryable without creating a GitHub branch.
  3. Use a direct provider and confirm OpenRouter-only routing settings are not sent upstream.

Visual Changes

N/A

Reviewer Notes

Provider-facing schemas remove unsupported rules, but the complete Zod schemas still validate every model reply inside the app. Failure reports contain only fixed messages, counts, and request IDs; they do not contain prompts, repository content, or raw model output.

Comment thread apps/web/src/lib/code-reviews/review-memory/change-request.ts Outdated
@kilo-code-bot

kilo-code-bot Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Focused refactor routes Review Memory model output through the AI SDK's native structured-output (Output.object with a provider-compatible JSON Schema) instead of manual JSON-text extraction, and adds OpenRouter require_parameters routing for structured-output requests; both are well-covered by tests.

Notes
  • generateReviewMemoryStructuredOutput (llm.ts) builds a wire JSON Schema from a Zod schema, strips Claude-incompatible keywords (transformReviewMemoryWireSchema), and lets the source Zod schema validate locally via { validate: sourceSchema.validate }. The contract previously restated in prompt text is now enforced at the provider boundary, so removing the inline JSON examples from buildReviewMemoryAnalysisPrompt / buildReviewMdIntegrationPrompt is safe.
  • applyOpenRouterStructuredOutputRouting only mutates body.provider for openrouter + chat_completions + json_schema requests, preserving existing provider fields via spread; the require_parameters?: boolean field is added to OpenRouterProviderConfig.
  • The prior inline comments about duplicated getSafeErrorClass are now moot — that helper and the surrounding diagnostics plumbing were removed entirely from llm.ts.
  • No memory leaks: structured-output helpers use per-call closures and module-scope pure constants only; the OpenRouter routing function only mutates its passed-in request.
Files Reviewed (7 files)
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.ts
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.test.ts
  • apps/web/src/lib/ai-gateway/providers/openrouter/types.ts
  • apps/web/src/lib/code-reviews/review-memory/aggregation.ts
  • apps/web/src/lib/code-reviews/review-memory/llm.ts
  • apps/web/src/lib/code-reviews/review-memory/llm.test.ts
  • apps/web/src/lib/code-reviews/review-memory/review-md-integration.ts
Previous Review Summaries (4 snapshots, latest commit af7f029)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit af7f029)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental commit removes redundant JSON-shape examples from two LLM prompts; structured output is already enforced via the JSON schema passed to the provider, so the removal is safe and reduces prompt token usage.

Incremental Changes (commit 8d2fcc4..HEAD)
  • apps/web/src/lib/code-reviews/review-memory/aggregation.ts — dropped the inline JSON example shapes from buildReviewMemoryAnalysisPrompt. The output contract is enforced by ReviewMemoryProposalDraftSchema / ReviewMemoryProposalWireSchema passed to generateReviewMemoryStructuredOutput (llm.ts:202-205 via Output.object({ schema: wireSchema })), so the prompt no longer needs to restate the shape.
  • apps/web/src/lib/code-reviews/review-memory/review-md-integration.ts — dropped the inline JSON shape from buildReviewMdIntegrationPrompt for the same reason; ReviewMdIntegrationOutputSchema enforces the contract at the provider boundary.

No tests assert on the removed prompt text, and both prompts still describe the required status values in their Rules sections (e.g. Return status "no_change" when ..., Return status "updated" with ...), so model behavior guidance is preserved.

Other Observations (not in diff)

Carried forward from prior reviews for the author's consideration (unchanged, non-blocking):

File Line Note
apps/web/src/lib/trpc/init.ts 21 REQUEST_CORRELATION_ID_PATTERN is strict; confirm it matches production Vercel IDs or every request silently falls back to a random UUID.
apps/web/src/lib/trpc/route-handler callers - Several createTRPCContext() callers pass no requestCorrelationId, so they won't share the Vercel trace ID.
apps/web/src/lib/code-reviews/review-memory/llm.ts - No memory leaks: structured output helper uses per-call closures and module-scope pure constants only; no module-scope client/transport caching.
Files Reviewed (2 files)
  • apps/web/src/lib/code-reviews/review-memory/aggregation.ts
  • apps/web/src/lib/code-reviews/review-memory/review-md-integration.ts

Previous review (commit 8d2fcc4)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental commit renames requireStructuredOutputParametersapplyOpenRouterStructuredOutputRouting consistently across implementation, caller, and tests; no new issues and prior DRY finding remains resolved.

Incremental Changes (commit 77a4160..HEAD)
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.ts — function renamed to better reflect its OpenRouter-specific structured-output routing behavior; call site updated.
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.test.ts — test describe block and call sites updated to the new name.

No remaining references to the old name found in the codebase. The new name is more accurate since the function only acts for the openrouter provider with json_schema response format.

Other Observations (not in diff)

Carried forward from prior reviews for the author's consideration (unchanged, non-blocking):

File Line Note
apps/web/src/lib/trpc/init.ts 21 REQUEST_CORRELATION_ID_PATTERN is strict; confirm it matches production Vercel IDs or every request silently falls back to a random UUID.
apps/web/src/lib/trpc/route-handler callers - Several createTRPCContext() callers pass no requestCorrelationId, so they won't share the Vercel trace ID.
apps/web/src/lib/code-reviews/review-memory/llm.ts - No memory leaks: no module-scope client caching, per-call closures, pure function references only.
Files Reviewed (2 files)
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.ts
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.test.ts

Previous review (commit 77a4160)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental commit 77a4160 resolves the prior DRY finding by exporting getSafeErrorClass from ./llm and importing it in change-request.ts; no new issues introduced.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 0

Incremental Changes (commit 77a4160)

  • apps/web/src/lib/code-reviews/review-memory/llm.tsgetSafeErrorClass is now exported (previously module-private).
  • apps/web/src/lib/code-reviews/review-memory/change-request.ts — removed the duplicated local getSafeErrorClass and imports it from ./llm instead. Verified the import is used at line 360.

The previously reported SUGGESTION (duplicate getSafeErrorClass) is now resolved.

Other Observations (not in diff)

Carried forward from the prior review for the author's consideration (unchanged, non-blocking):

File Line Note
apps/web/src/lib/trpc/init.ts 21 REQUEST_CORRELATION_ID_PATTERN is strict; confirm it matches production Vercel IDs or every request silently falls back to a random UUID.
apps/web/src/lib/trpc/route-handler callers - Several createTRPCContext() callers pass no requestCorrelationId, so they won't share the Vercel trace ID.
apps/web/src/lib/code-reviews/review-memory/llm.ts - No memory leaks: no module-scope client caching, per-call closures, pure function references only.
Files Reviewed (2 files)
  • apps/web/src/lib/code-reviews/review-memory/change-request.ts
  • apps/web/src/lib/code-reviews/review-memory/llm.ts

Previous review (commit 0d8d493)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Structured-output enforcement is well built with careful sanitization; the only finding is a minor DRY duplication of getSafeErrorClass across two new files.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
apps/web/src/lib/code-reviews/review-memory/change-request.ts 453 getSafeErrorClass is duplicated from ./llm.ts
Other Observations (not in diff)

No blocking issues. A few low-confidence notes for the author's consideration:

File Line Note
apps/web/src/lib/trpc/init.ts 21 REQUEST_CORRELATION_ID_PATTERN is very strict; if real x-vercel-id values don't match it, every request silently falls back to randomUUID() and won't correlate with Vercel request tracing. Functionally safe (random fallback), but worth confirming the pattern matches production Vercel IDs so the header is actually used.
apps/web/src/lib/trpc/route-handler callers - Several createTRPCContext() callers (trpc-route-handler.ts, code-indexing/* routes, cloud-agent-fork/review/[reviewId]/route.ts) pass no requestCorrelationId, so they always get a random UUID. Fine for correctness, but they won't share the Vercel trace ID if that's desired later.
apps/web/src/lib/code-reviews/review-memory/llm.ts - No memory leaks introduced: no module-scope caching of clients/promises, jsonSchema/onFinish closures are per-call, and defaultOperations holds only pure function references.
Files Reviewed (14 files)
  • apps/web/src/app/api/trpc/[trpc]/route.ts
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.test.ts
  • apps/web/src/lib/ai-gateway/providers/apply-provider-specific-logic.ts
  • apps/web/src/lib/ai-gateway/providers/openrouter/types.ts
  • apps/web/src/lib/code-reviews/review-memory/aggregation.test.ts
  • apps/web/src/lib/code-reviews/review-memory/aggregation.ts
  • apps/web/src/lib/code-reviews/review-memory/change-request.test.ts
  • apps/web/src/lib/code-reviews/review-memory/change-request.ts - 1 issue
  • apps/web/src/lib/code-reviews/review-memory/llm.test.ts
  • apps/web/src/lib/code-reviews/review-memory/llm.ts
  • apps/web/src/lib/code-reviews/review-memory/review-md-integration.test.ts
  • apps/web/src/lib/code-reviews/review-memory/review-md-integration.ts
  • apps/web/src/lib/trpc/init.ts
  • apps/web/src/routers/code-reviews/review-memory-router.ts
  • apps/web/src/routers/test-utils.ts

Fix these issues in Kilo Cloud


Reviewed by glm-5.2-20260616 · 1,135,271 tokens

Review guidance: REVIEW.md from base branch main


requestToMutate.body.provider = {
...requestToMutate.body.provider,
require_parameters: true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in what scenarios does setting this parameter make a difference?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants